-
Open
-
Open
-
Open
-
Open
Three Spanish MIT physics postdocs receive Botton Foundation fellowships
Recipients Luis Antonio Benítez, Carolina Cuesta-Lazaro, and Fernando Romero López receive support for their scientific research.
( 6
min )
-
Open
Host ML models on Amazon SageMaker using Triton: ONNX Models
ONNX (Open Neural Network Exchange) is an open-source standard for representing deep learning models widely supported by many providers. ONNX provides tools for optimizing and quantizing models to reduce the memory and compute needed to run machine learning (ML) models. One of the biggest benefits of ONNX is that it provides a standardized format for […]
( 14
min )
Fast-track graph ML with GraphStorm: A new way to solve problems on enterprise-scale graphs
We are excited to announce the open-source release of GraphStorm 0.1, a low-code enterprise graph machine learning (ML) framework to build, train, and deploy graph ML solutions on complex enterprise-scale graphs in days instead of months. With GraphStorm, you can build solutions that directly take into account the structure of relationships or interactions between billions […]
( 9
min )
-
Open
-
Open
The unannounced next-level partnership between Microsoft and Databricks
Microsoft publicly endorsed Open AI, with ‘Copilot’ embedded in every single bit of the Microsoft stack. Behind the scenes, with everything closed source, nobody knew if these AI assistants were driven by Cortana, Bing, or Open AI. The assistant technology is not new, and other than code generation and assisted writing, some wonder what value… Read More »The unannounced next-level partnership between Microsoft and Databricks
The post The unannounced next-level partnership between Microsoft and Databricks appeared first on Data Science Central.
( 22
min )
The Importance of Data Engineering for a Profitable App Development
High-quality app development can significantly drive your business growth and success while boosting customer satisfaction and bringing in more clients. However, with millions of apps existing in the market, standing out from the competition requires more than just a great idea and an appealing design. Data engineering is what can help you, playing a pivotal… Read More »The Importance of Data Engineering for a Profitable App Development
The post The Importance of Data Engineering for a Profitable App Development appeared first on Data Science Central.
( 21
min )
-
Open
Eye in the Sky With AI: UCSB Initiative Aims to Pulverize Space Threats Using NVIDIA RTX
When meteor showers occur every few months, viewers get to watch a dazzling scene of shooting stars and light streaks scattering across the night sky. Normally, meteors are just small pieces of rock and dust from space that quickly burn up upon entering Earth’s atmosphere. But the story would take a darker turn if a Read article >
( 7
min )
-
Open
-
Open
AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation. (arXiv:2210.07535v2 [cs.CL] UPDATED)
Mixture-of-Expert (MoE) models have obtained state-of-the-art performance in
Neural Machine Translation (NMT) tasks. Existing works in MoE mostly consider a
homogeneous design where the same number of experts of the same size are placed
uniformly throughout the network. Furthermore, existing MoE works do not
consider computational constraints (e.g., FLOPs, latency) to guide their
design. To this end, we develop AutoMoE -- a framework for designing
heterogeneous MoE's under computational constraints. AutoMoE leverages Neural
Architecture Search (NAS) to obtain efficient sparse MoE sub-transformers with
4x inference speedup (CPU) and FLOPs reduction over manually designed
Transformers, with parity in BLEU score over dense Transformer and within 1
BLEU point of MoE SwitchTransformer, on aggregate over benchmark datasets for
NMT. Heterogeneous search space with dense and sparsely activated Transformer
modules (e.g., how many experts? where to place them? what should be their
sizes?) allows for adaptive compute -- where different amounts of computations
are used for different tokens in the input. Adaptivity comes naturally from
routing decisions which send tokens to experts of different sizes. AutoMoE
code, data, and trained models are available at https://aka.ms/AutoMoE.
( 2
min )
Sequence-to-Sequence Model with Transformer-based Attention Mechanism and Temporal Pooling for Non-Intrusive Load Monitoring. (arXiv:2306.05012v1 [eess.SP])
This paper presents a novel Sequence-to-Sequence (Seq2Seq) model based on a
transformer-based attention mechanism and temporal pooling for Non-Intrusive
Load Monitoring (NILM) of smart buildings. The paper aims to improve the
accuracy of NILM by using a deep learning-based method. The proposed method
uses a Seq2Seq model with a transformer-based attention mechanism to capture
the long-term dependencies of NILM data. Additionally, temporal pooling is used
to improve the model's accuracy by capturing both the steady-state and
transient behavior of appliances. The paper evaluates the proposed method on a
publicly available dataset and compares the results with other state-of-the-art
NILM techniques. The results demonstrate that the proposed method outperforms
the existing methods in terms of both accuracy and computational efficiency.
( 2
min )
BTS: Bifold Teacher-Student in Semi-Supervised Learning for Indoor Two-Room Presence Detection Under Time-Varying CSI. (arXiv:2212.10802v3 [cs.AI] UPDATED)
In recent years, indoor human presence detection based on supervised learning
(SL) and channel state information (CSI) has attracted much attention. However,
existing studies that rely on spatial information of CSI are susceptible to
environmental changes which degrade prediction accuracy. Moreover, SL-based
methods require time-consuming data labeling for retraining models. Therefore,
it is imperative to design a continuously monitored model using a
semi-supervised learning (SSL) based scheme. In this paper, we conceive a
bifold teacher-student (BTS) learning approach for indoor human presence
detection in an adjoining two-room scenario. The proposed SSL-based primal-dual
teacher-student network intelligently learns spatial and temporal features from
labeled and unlabeled CSI datasets. Additionally, the enhanced penalized loss
function leverages entropy and distance measures to distinguish drifted data,
i.e., features of new datasets affected by time-varying effects and altered
from the original distribution. Experimental results demonstrate that the
proposed BTS system sustains asymptotic accuracy after retraining the model
with unlabeled data. Furthermore, BTS outperforms existing SSL-based models in
terms of the highest detection accuracy while achieving the asymptotic
performance of SL-based methods.
( 2
min )
Efficient Computation of Shap Explanation Scores for Neural Network Classifiers via Knowledge Compilation. (arXiv:2303.06516v2 [cs.AI] UPDATED)
The use of Shap scores has become widespread in Explainable AI. However,
their computation is in general intractable, in particular when done with a
black-box classifier, such as neural network. Recent research has unveiled
classes of open-box Boolean Circuit classifiers for which Shap can be computed
efficiently. We show how to transform binary neural networks into those
circuits for efficient Shap computation. We use logic-based knowledge
compilation techniques. The performance gain is huge, as we show in the light
of our experiments.
( 2
min )
A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models. (arXiv:2210.12023v3 [cs.CL] UPDATED)
We have recently witnessed a number of impressive results on hard
mathematical reasoning problems with language models. At the same time, the
robustness of these models has also been called into question; recent works
have shown that models can rely on shallow patterns in the problem description
when generating a solution. Building on the idea of behavioral testing, we
propose a novel framework, which pins down the causal effect of various factors
in the input, e.g., the surface form of the problem text, the operands, and
math operators on the output solution. By grounding the behavioral analysis in
a causal graph describing an intuitive reasoning process, we study the behavior
of language models in terms of robustness and sensitivity to direct
interventions in the input space. We apply our framework on a test bed of math
word problems. Our analysis shows that robustness does not appear to
continuously improve as a function of size, but the GPT-3 Davinci models (175B)
achieve a dramatic improvement in both robustness and sensitivity compared to
all other GPT variants.
( 2
min )
Deploying clinical machine learning? Consider the following.... (arXiv:2109.06919v3 [cs.LG] UPDATED)
Despite the intense attention and considerable investment into clinical
machine learning research, relatively few applications have been deployed at a
large-scale in a real-world clinical environment. While research is important
in advancing the state-of-the-art, translation is equally important in bringing
these techniques and technologies into a position to ultimately impact
healthcare. We believe a lack of appreciation for several considerations are a
major cause for this discrepancy between expectation and reality. To better
characterize a holistic perspective among researchers and practitioners, we
survey several practitioners with commercial experience in developing CML for
clinical deployment. Using these insights, we identify several main categories
of challenges in order to better design and develop clinical machine learning
applications.
( 2
min )
Interpreting and Improving Diffusion Models Using the Euclidean Distance Function. (arXiv:2306.04848v1 [cs.LG])
Denoising is intuitively related to projection. Indeed, under the manifold
hypothesis, adding random noise is approximately equivalent to orthogonal
perturbation. Hence, learning to denoise is approximately learning to project.
In this paper, we use this observation to reinterpret denoising diffusion
models as approximate gradient descent applied to the Euclidean distance
function. We then provide straight-forward convergence analysis of the DDIM
sampler under simple assumptions on the projection-error of the denoiser.
Finally, we propose a new sampler based on two simple modifications to DDIM
using insights from our theoretical results. In as few as 5-10 function
evaluations, our sampler achieves state-of-the-art FID scores on pretrained
CIFAR-10 and CelebA models and can generate high quality samples on latent
diffusion models.
( 2
min )
Fast and Effective GNN Training with Linearized Random Spanning Trees. (arXiv:2306.04828v1 [cs.LG])
We present a new effective and scalable framework for training GNNs in
supervised node classification tasks, given graph-structured data. Our approach
increasingly refines the weight update operations on a sequence of path graphs
obtained by linearizing random spanning trees extracted from the input network.
The path graphs are designed to retain essential topological and node
information of the original graph. At the same time, the sparsity of path
graphs enables a much lighter GNN training which, besides scalability, helps in
mitigating classical training issues, like over-squashing and over-smoothing.
We carry out an extensive experimental investigation on a number of real-world
graph benchmarks, where we apply our framework to graph convolutional networks,
showing simultaneous improvement of both training speed and test accuracy, as
compared to well-known baselines.
( 2
min )
Privately generating tabular data using language models. (arXiv:2306.04803v1 [cs.LG])
Privately generating synthetic data from a table is an important brick of a
privacy-first world. We propose and investigate a simple approach of treating
each row in a table as a sentence and training a language model with
differential privacy. We show this approach obtains competitive results in
modelling tabular data across multiple datasets, even at small scales that
favor alternative methods based on marginal distributions.
( 2
min )
Embedding stochastic differential equations into neural networks via dual processes. (arXiv:2306.04847v1 [cs.LG])
We propose a new approach to constructing a neural network for predicting
expectations of stochastic differential equations. The proposed method does not
need data sets of inputs and outputs; instead, the information obtained from
the time-evolution equations, i.e., the corresponding dual process, is directly
compared with the weights in the neural network. As a demonstration, we
construct neural networks for the Ornstein-Uhlenbeck process and the noisy van
der Pol system. The remarkable feature of learned networks with the proposed
method is the accuracy of inputs near the origin. Hence, it would be possible
to avoid the overfitting problem because the learned network does not depend on
training data sets.
( 2
min )
Special Session: Approximation and Fault Resiliency of DNN Accelerators. (arXiv:2306.04645v1 [cs.LG])
Deep Learning, and in particular, Deep Neural Network (DNN) is nowadays
widely used in many scenarios, including safety-critical applications such as
autonomous driving. In this context, besides energy efficiency and performance,
reliability plays a crucial role since a system failure can jeopardize human
life. As with any other device, the reliability of hardware architectures
running DNNs has to be evaluated, usually through costly fault injection
campaigns. This paper explores the approximation and fault resiliency of DNN
accelerators. We propose to use approximate (AxC) arithmetic circuits to
agilely emulate errors in hardware without performing fault injection on the
DNN. To allow fast evaluation of AxC DNN, we developed an efficient GPU-based
simulation framework. Further, we propose a fine-grain analysis of fault
resiliency by examining fault propagation and masking in networks
( 2
min )
-
Open
Causal Inference of General Treatment Effects using Neural Networks with A Diverging Number of Confounders. (arXiv:2009.07055v6 [stat.ME] UPDATED)
The estimation of causal effects is a primary goal of behavioral, social,
economic and biomedical sciences. Under the unconfoundedness condition,
adjustment for confounders requires estimating the nuisance functions relating
outcome and/or treatment to confounders. This paper considers a generalized
optimization framework for efficient estimation of general treatment effects
using feedforward artificial neural networks (ANNs) when the number of
covariates is allowed to increase with the sample size. We estimate the
nuisance function by ANNs, and develop a new approximation error bound for the
ANNs approximators when the nuisance function belongs to a mixed Sobolev space.
We show that the ANNs can alleviate the curse of dimensionality under this
circumstance. We further establish the consistency and asymptotic normality of
the proposed treatment effects estimators, and apply a weighted bootstrap
procedure for conducting inference. The proposed methods are illustrated via
simulation studies and a real data application.
( 2
min )
DP-Fast MH: Private, Fast, and Accurate Metropolis-Hastings for Large-Scale Bayesian Inference. (arXiv:2303.06171v2 [cs.LG] UPDATED)
Bayesian inference provides a principled framework for learning from complex
data and reasoning under uncertainty. It has been widely applied in machine
learning tasks such as medical diagnosis, drug design, and policymaking. In
these common applications, data can be highly sensitive. Differential privacy
(DP) offers data analysis tools with powerful worst-case privacy guarantees and
has been developed as the leading approach in privacy-preserving data analysis.
In this paper, we study Metropolis-Hastings (MH), one of the most fundamental
MCMC methods, for large-scale Bayesian inference under differential privacy.
While most existing private MCMC algorithms sacrifice accuracy and efficiency
to obtain privacy, we provide the first exact and fast DP MH algorithm, using
only a minibatch of data in most iterations. We further reveal, for the first
time, a three-way trade-off among privacy, scalability (i.e. the batch size),
and efficiency (i.e. the convergence rate), theoretically characterizing how
privacy affects the utility and computational cost in Bayesian inference. We
empirically demonstrate the effectiveness and efficiency of our algorithm in
various experiments.
( 2
min )
Exact Optimality of Communication-Privacy-Utility Tradeoffs in Distributed Mean Estimation. (arXiv:2306.04924v1 [cs.LG])
We study the mean estimation problem under communication and local
differential privacy constraints. While previous work has proposed
\emph{order}-optimal algorithms for the same problem (i.e., asymptotically
optimal as we spend more bits), \emph{exact} optimality (in the non-asymptotic
setting) still has not been achieved. In this work, we take a step towards
characterizing the \emph{exact}-optimal approach in the presence of shared
randomness (a random variable shared between the server and the user) and
identify several necessary conditions for \emph{exact} optimality. We prove
that one of the necessary conditions is to utilize a rotationally symmetric
shared random codebook. Based on this, we propose a randomization mechanism
where the codebook is a randomly rotated simplex -- satisfying the necessary
properties of the \emph{exact}-optimal codebook. The proposed mechanism is
based on a $k$-closest encoding which we prove to be \emph{exact}-optimal for
the randomly rotated simplex codebook.
( 2
min )
